Graphics remoting systems allow computing device network clients to connect to a server and receive a visual representation of at least some of the graphics being displayed at or output by the server. Often the network client can display all the graphical output associated with the session. Likewise, the client may be allowed to interact with the session, injecting user input, generated from devices such as a mouse or keyboard connected to the client, into the server session. Such a server that allows a client to connect in a “logon session” is sometimes referred to as a “remote server” or a “remoting server.” Computing device subsystems that execute remote logon sessions and graphics remoting are sometimes called “terminal services subsystems.”
A graphics remoting system can control the amount of graphics sent to a client so that the client does not see everything being displayed on (i.e., “output by”) a remoting server. Three categories conveniently divide the degree of graphics sharing between server and client. In “desktop remoting,” a client sees substantially everything that would be seen on the canvas of a display monitor connected directly to the server. In “window remoting,” a client sees only a window or pane (or a set of windows or panes) from the server's desktop, each window usually projected by the server's operating system for the graphical output and/or user interface (UI) corresponding to one application. Hence, at the server a word processor application might own one window and an email application might own another window, but perhaps only the word processor window would be shared with the client. In “region remoting,” only a subset of the server's desktop is shared with a client, but the region shared does not necessarily correspond to a window. In fact, the shared region may be arbitrary.
In order to accomplish this graphical remoting “action-at-a-distance” between a remoting server and a logged-on client, a certain amount of telemetry is needed. From the perspective of the client, an instrumentation at the server side records the “graphics data,” that is, information that makes up the visual content of the desktop or a region of the desktop, and sends this graphics data back to the client. The graphics data may describe text, image content, and/or UI features, such as scroll bars and clickable buttons being generated by the server. The graphics data alone may be sufficient information for the above-described desktop remoting, but insufficient for window and region types of remoting. In these latter types of remoting, a second instrumentation at the server side must measure the geometry (i.e., the shape) of a region to be shared, and transmit this geometry information back to the client. The second instrumentation or yet another third instrumentation at the server side must also measure the relative placement of the region to be shared with respect to the server's desktop (hereinafter called “position”) and transmit this third type of information back to the client. Thus, to accurately display a region from the server's desktop on its own display, the client must be informed of the region's current graphics data, shape, and position (the latter two being “region data.”) The client's view of the region data—the shape and position information—allows the client to divide the server desktop into visible and non-visible regions, i.e., region(s) on which to display associated graphics data and region(s) on which to not display associated graphics data. The client then fills the region(s) to display with the graphics data, or at least clips the graphics data with the boundaries of the region(s) not to display.
Conventionally, the above-mentioned graphics data of a region to be shared is sensed, tracked, and/or gathered continuously by a first instrumentation and transmitted to the client independently of the shape and position information. This first instrumentation usually consists of or relies heavily on well-evolved parts of an operating system's kernel graphics processing subsystem, since much of the graphics tracking is the same as might be used if the server were not a remoting server.
The region data are conventionally gathered in an asynchronous manner by a second instrumentation that is on a completely different scheduler than the kernel components that continuously gather the graphics data. For example, region data may be gathered in user mode, by polling mechanisms. Moreover, the region data are sent to the client desynchronized from the graphics data or, in other words, if the graphics data and region data are combined in a single data stream by a remoting protocol, the combination and adherence to remoting protocol rules does not cure the asynchronicity of the region data with respect to the graphics data. (Asynchronous collection of region data means, for one thing, that the region data collection operates independently of the graphics data collection, i.e., without reference to matching a segment of the graphics data with a segment of the region data.)
The received graphics data and region data are fused together by the client to create the final display. Historically, using two instrumentations working more or less independently of each other to asynchronously collect the graphics data and the region data evolved perhaps because mechanisms in operating system kernels for collecting graphics data were already well-evolved when remoting began to be practiced, and collection of the region data was added as a latecomer or an afterthought. Since they are latecomers, methods for asynchronously collecting region data typically avoid disturbing the operating system kernel, which is the realm of the graphics data collection.
Once graphics data and region data are asynchronously collected for region remoting, they are typically packetized according to a protocol, such as MICROSOFT® REMOTE DESKTOP PROTOCOL (RDP) or CITRIX INDEPENDENT COMPUTING ARCHITECTURE (ICA) PROTOCOL, and sent to the client. (Microsoft Corporation, Redmond, Wash.; Citrix Corporation, Fort Lauderdale, Fla.) The RDP protocol, for example, is just one of many graphics remoting protocols, that is, many graphics remoting protocols are extensible enough to add region remoting for sharing a window or a region of a desktop. If RDP is used as a data transmission component of the remote display telemetry, then it should be noted that RDP is based on, and is an extension of, the T.120 protocol family standards. RDP is a multichannel-capable protocol that allows for separate virtual channels to carry device communication and presentation data from the server, as well as encrypted client mouse and keyboard data.
Conventionally, the graphics data and the region data are desynchronized at their inception by asynchronous collection and may become even more desynchronized depending on how the data is handled as it passes through network layers to be transmitted to and received by a client. The RDP protocol does not cure the desynchronization. By analogy, like a bird watcher whose tracking of a quick pheasant with a pair of excellent binoculars does not stay synchronized (despite the excellence of the binoculars), a remoting client that receives desynchronized graphics and region data (despite the excellence of the remoting protocol), not only misses seeing a desired target on the server desktop but may instead actually see things on the server desktop that he is not supposed to see.
FIG. 1 shows parts of a graphics remoting system 100, including a server display 102 and a client display 104. A window 106 to be shared with the client displays on the server display 102 together with “forbidden visual regions” of secret and private information 108 on the server's desktop. If at any point in time, the graphics data describing the content of the window 106 and the region data describing the shape and placement of the window 106 are not synchronized, the client display 104 will show the intended window 106′ incorrectly. For example, if the window 106′ moves or is moved from left to right across the desktop of the server display 102 and the graphics data and region data are not synchronized when collected and/or when transmitted to the client, then the client display 104 may continue to display that region of the server desktop where the window was originally positioned 110, but which is now displaying secret information.
This desynchronization causes two problems: not displaying content that should be presented, and displaying content that should not be presented. First, the client display 104 does not show the subset of the server display 102 that the graphics remoting system 100 intends to show the client. That is, if the window 106 intended to be displayed contains data necessary to make business decisions, then the decisions are stalled until correct data is displayed. Second, the client display 104 may show, albeit fleetingly, unintended secret information 112, i.e., a subset of the server display 102 that the graphics remoting system 100 does not intend to share with the client. Hence, the desynchronization presents a security risk if “top secret” information is revealed to a client without the proper clearance. Unfortunately, synchronization problems between graphics data and region data exist in most known graphics remoting systems, wherein clients often see unintended parts of a server desktop.